Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary by kala-moz · Pull Request #1034 · mozilla/perfcompare

kala-moz · 2026-05-05T00:31:09Z

This PR (Bug 2036968) integrates the js version of @padenot's standalone-stats into perfcompare. Here are the main changes for the integration:

Deploy Link

1.) Added standalone-stats/kde.js and kde.d.ts; bootstrap-ci.js and bootstrap-ci.ts to src/utils
2.) In src/components/CompareResults/CommonGraph.tsx - Removed the the fast-kde import and replaced with the fftkde. I removed approximateSJBandwidth because the ISJ method below auto-selects the bandwidth per dataset. It also has an internal grid and quantile logic so didn't need the quantileSorted method.

  const bKde =
    baseValues.length >= 2 ? fftkde(baseValues, 'ISJ', undefined, 1024) : null;
  const nKde =
    newValues.length >= 2 ? fftkde(newValues, 'ISJ', undefined, 1024) : null;
  const baseRunsDensity = bKde
    ? bKde.x.map((x, i) => ({ x, y: bKde.y[i] }))
    : [];
  const newRunsDensity = nKde
    ? nKde.x.map((x, i) => ({ x, y: nKde.y[i] }))
    : [];

3.) Use the bootstrapMedianDiffCI from bootstrap-ci to get the confidence interval summary and confidence interval statement.

   const baseRuns = mwResult.base_runs ?? [];
    const newRuns = mwResult.new_runs ?? [];
    const ci =
      baseRuns.length > 0 && newRuns.length > 0
        ? bootstrapMedianDiffCI(baseRuns, newRuns)
        : null;
    const fmt = (n: number) => (n >= 0 ? '+' : '') + n.toFixed(1);
    const summary = ci ? (
      <span>
        <strong>Δ median</strong> = {fmt(ci.medianDiff)} ms 95% CI [
        {fmt(ci.ciLow)}, {fmt(ci.ciHigh)}]
        {ci.significant ? '' : '  (not significant)'}
      </span>
    ) : null;
    const confidenceInterval = ci && (
      <span>
        <strong>Confidence Interval</strong>: We are 95% confident the median
        difference is between <strong>{fmt(ci.ciLow)}</strong> and{' '}
        <strong>{fmt(ci.ciHigh)}</strong>
      </span>
    );

Note: please check that the comments are relevant as I may have missed to update or add them where necessary.
Still need to add coverage.

Updates:

Hook up CommonGraph to echarts, replacing chartjs

netlify · 2026-05-05T00:31:14Z

✅ Deploy Preview for mozilla-perfcompare ready!

Name	Link
🔨 Latest commit	`ce542fe`
🔍 Latest deploy log	https://app.netlify.com/projects/mozilla-perfcompare/deploys/6a0f3f28923fcb0008c40727
😎 Deploy Preview	https://deploy-preview-1034--mozilla-perfcompare.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2026-05-05T00:32:50Z

Codecov Report

❌ Patch coverage is 98.97959% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.56%. Comparing base (c38aace) to head (ce542fe).

Files with missing lines	Patch %	Lines
src/components/CompareResults/CommonGraph.tsx	98.78%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1034      +/-   ##
==========================================
+ Coverage   96.38%   96.56%   +0.17%     
==========================================
  Files         113      113              
  Lines        3267     3288      +21     
  Branches      736      746      +10     
==========================================
+ Hits         3149     3175      +26     
+ Misses        117      112       -5     
  Partials        1        1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

padenot · 2026-05-07T13:35:14Z

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

kala-moz · 2026-05-08T00:00:07Z

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

Thanks, @padenot. I went ahead and switched to echarts. Should I continue with the rest of the updates or should we break them into separate PRs? I feel this PR is pretty full already.

padenot

As discussed, those are only some changes to do now, and we'll do the rest in a second PR. Looks nice, most comments are cosmetics or just comments (not linked to any action).

kala-moz · 2026-05-13T20:22:29Z

Found a bug in my patch. The graph breaks with this comparison on some tests. https://deploy-preview-1034--mozilla-perfcompare.netlify.app/compare-results?baseRev=422aa058a4418b51b7cbccc3a6b3cc0e5f4c5d5b&baseRepo=autoland&newRev=b2144d802d0ad9cf9d259ca0cf62f9ec9b3615ec&newRepo=autoland&framework=13&sort=delta%7Cdesc

… bootstrap-ci

…ow both densities at a single x position Updated tests to reflect changes

…man fallback, ref bail-out, and tooltip formatter; also improved x-axis label

kala-moz · 2026-05-19T20:13:19Z

Found a bug in my patch. The graph breaks with this comparison on some tests. https://deploy-preview-1034--mozilla-perfcompare.netlify.app/compare-results?baseRev=422aa058a4418b51b7cbccc3a6b3cc0e5f4c5d5b&baseRepo=autoland&newRev=b2144d802d0ad9cf9d259ca0cf62f9ec9b3615ec&newRepo=autoland&framework=13&sort=delta%7Cdesc

Bug fixed.

padenot

Looks good. Look for follow-up for a couple things we discussed on the channels.

This reverts commit d19a621.

kala-moz requested review from beatrice-acasandrei and gmierz May 5, 2026 00:35

kala-moz changed the title ~~Integrate js stats analysis~~ Bug 2036968: Integrate js stats analysis May 5, 2026

kala-moz requested a review from padenot May 6, 2026 20:30

kala-moz changed the title ~~Bug 2036968: Integrate js stats analysis~~ Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary May 7, 2026

kala-moz marked this pull request as draft May 7, 2026 16:22

kala-moz marked this pull request as ready for review May 8, 2026 00:45

kala-moz requested review from canova and esanuandra as code owners May 8, 2026 00:45

padenot approved these changes May 11, 2026

View reviewed changes

kala-moz added 10 commits May 18, 2026 19:30

use padenots kde for commongraph and get the confidence interval from…

ed94b42

… bootstrap-ci

replaced chartjs with echarts

96f2203

update package.json

5793318

update lockfile

7413ba8

add test coverage; fix lockfile

6e1e556

fall back to Silverman when ISJ bandwidth fails to converge

2c3e6cb

don't display CommonGraph if either baseValues or newValues are missing

186f549

extract 1024 as global and remove scatter series and related tests

70ddfdc

resample base and new KDEs onto a shared x-grid so the tooltip can sh…

61e4ab9

…ow both densities at a single x position Updated tests to reflect changes

update test merge conflict

22dafeb

kala-moz force-pushed the js-integration-analysis branch from f2bb4a7 to 22dafeb Compare May 19, 2026 18:39

add CommonGraph.test.tsx with focused coverage for resampling, Silver…

4654672

…man fallback, ref bail-out, and tooltip formatter; also improved x-axis label

round x-axis tick labels to 2 dp, leaving whole numbers as bare integers

318bda9

kala-moz removed the request for review from canova May 20, 2026 16:57

kala-moz removed the request for review from esanuandra May 20, 2026 16:57

padenot approved these changes May 21, 2026

View reviewed changes

Comment thread src/components/CompareResults/CommonGraph.tsx Outdated

Comment thread src/components/CompareResults/CommonGraph.tsx Outdated

Comment thread src/utils/bootstrap-ci.js Outdated

Comment thread package-lock.json

Comment thread src/__tests__/CompareResults/ResultsView.test.tsx Outdated

kala-moz added 3 commits May 21, 2026 10:07

delete bootstrap-ci.js

2e02b5b

revert package-lock.json to original form

d19a621

Revert "revert package-lock.json to original form"

ce542fe

This reverts commit d19a621.

kala-moz merged commit bb77ec5 into mozilla:main May 21, 2026
10 checks passed

Conversation

kala-moz commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Updates:

Uh oh!

netlify Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for mozilla-perfcompare ready!

Uh oh!

codecov Bot commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

padenot commented May 7, 2026

Uh oh!

kala-moz commented May 8, 2026

Uh oh!

padenot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kala-moz commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kala-moz commented May 19, 2026

Uh oh!

padenot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kala-moz commented May 5, 2026 •

edited

Loading

netlify Bot commented May 5, 2026 •

edited

Loading

codecov Bot commented May 5, 2026 •

edited

Loading

kala-moz commented May 13, 2026 •

edited

Loading